System for organizing and visualizing display objects

ABSTRACT

A method, system and computer program for organizing and visualizing display objects within a virtual environment is provided. In one aspect, attributes of display objects define the interaction between display objects according to pre-determined rules, including rules simulating real world mechanics, thereby enabling enriched user interaction. The present invention further provides for the use of piles as an organizational entity for desktop objects. The present invention further provides for fluid interaction techniques for committing actions on display objects in a virtual interface. A number of other interaction and visualization techniques are disclosed.

This application claims the benefit of U.S. Provisional Application No.60/793,630 filed on 21 Apr. 2006.

FIELD OF THE INVENTION

The present invention relates to methods of organizing and visualizingdisplay objects within virtual environments. In particular, the presentinvention relates to interaction and visualization techniques fororganizing display objects within virtual environments.

BACKGROUND OF THE INVENTION

Despite the metaphor, current virtual desktops bear little resemblanceto the look or feel of real world desktops. A workspace in the physicalworld typically has piles of documents, binders and other objectsarranged in a way that provides considerable subtle information to theowner. For example, items are often casually placed but their spatialposition and orientation are usually meaningful. Closer items canindicate urgency, and piles of items are “automatically” orderedchronologically because new items are typically placed on top. Thiscasual organization, prevalent in the real world, differs greatly fromthe GUI desktop which forces users to immediately file their documentsinto a rigid hierarchy. Filing typically requires more effort thanpiling and has been shown to have other negative effects such asencouraging premature storage of low value documents, or retaininguseless documents because of the effort that went into filing them.

There has been significant research and development in this area. Officeworker organizational behaviour studies have identified two generalpaper organization strategies: ‘piling’ and ‘filing’. It has also beenfound that categorizing and filing items was cognitively difficult. Ithas been noted that virtual desktops should provide untitled piles thatsupport deferred classification as well as titled, logically arrangedfiles. Further, it has been postulated that ‘electronic piles’ shouldmake the use of computers more natural.

Piling as an organizing strategy has several advantages over filing.(Whittaker, S. & Hirschberg, J. (2001), The character, value, andmanagement of personal paper archives, ACM Trans on CHI, 8(2), p.150-170.) Piling is lightweight, casual, involves less overhead, and iseasier to maintain than filing. Piles serve as visual reminders andincreased availability of recent information. Pilers more frequentlyaccess their piles than filers accessed their file archives. Pilersarchives may also be smaller, which can be attributed to piledinformation being easier to discard. Filers reluctantly discardinformation due to the effort put into initially filing it. Filers alsoprematurely filed documents later deemed to be of little or no value. Inaddition, sometimes more than one filing category applies, or anexisting category is forgotten and a new one created. On the other hand,piling did not scale well and information was difficult to find once thenumber of piles grew large. Taken to excess, piling can take over everysurface in an office. Despite the advantages of piling, there remainslittle technological support for piling in today's GUI desktops.

The pile metaphor has been explored in a prototype. (Mander, R.,Salomon, G., & Wong, Y. (1992), A “pile” metaphor for supporting casualorganization of information, CHI, p. 260-269.) The prototype was basedon a user-centered iterative design process. Gestures and interactiontechniques were introduced (sometimes modal) for browsing andmanipulating piles and facilitating ‘casual organization’ on thedesktop.

Previous work has also looked at piles in different contexts. DiGioia etal. used a pile visualization to aid ‘social navigation’ and security.(DiGioia, P. & Dourish, P. (2005), Social navigation as a model forusable security, ACM SOUPS, p. 101-108.) Ruffled piles were used toindicate information a group of users frequently accessed. To removedocuments or piles from public access they could be moved to a ‘filingcabinet’.

DynaPad™'s “open” pile representation laid out entire collections ofphotos side-byside on a zoomable Pad++™ based workspace. (Bauer, D.,Fastrez, P., & Hollan, J. (2004), Computationally-enriched “piles” formanaging digital photo collections, IEEE VLHCC, p. 193-195; Bederson, B.& Hollan, J. (1994), Pad++: a zooming graphical interface for exploringalternate interface physics, UIST, p. 17-26.) This representation avoidsocclusion of stacked items with each other, but results in higher visualload and greater screen real-estate requirements. The latter issue ismitigated because of infinite canvas. “Open” piles also aim to enhanceremindability through visibility of all sub-objects, although thisdiminishes when the workspace is zoomed out and thumbnails become small.The alternative stack representation is in the spirit of familiarreal-world piles and does not require a zoomable interface. The linearlyordered piles support fluid sorting and re-ordering in place without theneed for additional tools.

Recent physically-inspired GUI designs rethink windows as paper stackedin piles. Windows can be freeform peeled like real pieces of paper witha robust algorithm. Peeling and re-orientation allows viewing ofoccluded windows below. (Beaudouin-Lafon, M. (2001), Novel interactiontechniques for overlapping windows, UIST, p. 152-154.)

Denoue et al. disclosed using real-time simulated cloth texture-mappedas fliers and pinned up to virtual bulletin board that blow in the wind.(Denoue, L., Nelson, L., & Churchill, E. (2003), A fast, interactive 3Dpaper-flier metaphor for digital bulletin boards, UIST, p. 169-172.)

Tossing as a window moving technique has also been disclosed. (Yatani,K., Tamura, K., Hiroki, K., Sugimoto, M., & Hasizume, H. (2005),Toss-it: intuitive information transfer techniques for mobile devices,CHI Ext. Abs., p. 1881-1884; Streitz, N., Geiβler, J., Holmer, T.,Konomi, S. i., Müller-Tomfelde, C., Reischl, W., Rexroth, P., Seitz, P.,& Steinmetz, R. (1999), i-LAND: an interactive landscape for creativityand innovation, CHI, p. 120-127.)

The benefits of spatially based organization have also been shown.Leveraging spatial memory in organizing webpage thumbnails on aperspective 2½D plane has showed improved user performance againsttext-based webpage bookmarks. (Robertson, G., Czerwinski, M., Larson,K., Robbins, D., Thiel, D., & van Dantzich, M. (1998), Data mountain:Using spatial memory for document management, UIST, p. 153-162.)

Recent investigations into pen-based computing have broken away fromtraditional point-and-click interfaces to techniques that are easieraccomplished with the pen such as goal crossing. (Accot, J. & Zhai, S.(2002), More than dotting the i's—foundations for crossing-basedinterfaces, CHI, p. 73-80.)

In addition to the academic developments and disclosures, there are somenotable patents in this field. U.S. Pat. No. 5,303,388 to Kreitman etal. describes manipulable icons that are represented asthree-dimensional objects with different data on each side that a usercan manipulate. A user is therefore required to manually rotate icons toview the additional information. In addition, representing informationon every side of the object constrain the potential shape of the objectin order to maintain readability of each of the sides. Representinginformation on each side of an icon can also waste screen real estate asthere may not be any relevant information to display for particularobjects.

U.S. Pat. Nos. 5,838,326 and 5,847,709 to Card et al. describes a “flickgesture” to move documents around to different components of anelectronic workspace. The system is limited because it allows thedocuments to be tossed to finite areas only. “Flick gestures” in thecardinal directions are mapped to one-to-one onto specific workspaceareas. This limits the expressive range and number of places documentsmay be moved with this technique. As well, the use of time-based gesturedetection has inherent problems in recognition resulting in inaccuracyand false positives.

In U.S. Pat. No. 6,677,965 to Ullmann et al. a virtual rubber-band iscreated between the cursor and a GUI control such as a slider orscrollbar. This provides visual feedback and variable rate control ofthe slider dependent on how the cursor is moved from the GUI control.The technique is only applied to discrete 1-dimensional positioning ofGUI controls.

U.S. Pat. No. 6,915,489 to Gargi discloses a system that stacks imagesdiagonally and allows moving the mouse in a direction to browse them ata disjoint screen location. This diagonal arrangement with disjointsecondary-image requires a large amount of screen real estate and doesnot scale well to a very large number of items. The diagonal layout alsodoes not efficiently use the space to the top-right or bottom-left ofthe diagonal arrangement of images. Also, this stacking techniqueoccludes much of the images with the ones on top of it.

U.S. Pat. Nos. 6,928,621, 5,583,984 and 6,307,545 to Conrad et al.describe a “spring loaded folders” technique. Moving the mouse andpausing over an “enclosure” opens its window temporarily while the mouseis down. The new window may occlude items behind it, and browsing willrequire moving the mouse cursor outside the layers of sprung openwindows, and then reacquisition of potentially occluded other folders.

U.S. Pat. No. 6,907,580 to Michelman et al. describes a moveable userinterface element containing displayed options for further manipulationof a selected object. However, user interface elements are triggered bypoint-and-click interaction, which breaks user flow and is notcontinuous.

In addition, U.S. Pat. Nos. 6,613,101 and 6,243,724 to Mander et al.discloses a mode-based piling approach to document organization.Gestures and interaction techniques were disclosed for browsing andmanipulating piles and “casual organization” on a desktop. However,mode-based interaction techniques are known to be problematic as usersoften forget what mode they are in or that they need to switch. Inaddition, their approach uses idiosyncratic gestures which are prone toimperfect recognition and requiring memorization are used to triggerinteraction. Also, some of the techniques disclosed are in isolation andnot integrated with each other. For instance, sorting piles required aspecial mode with its own interface inside a dialog box, and is notintegrated with the main display.

In light of the foregoing, what is needed is an improved method, systemand computer program for organizing and visualizing display objectswithin a virtual environment. In particular, what is needed is a method,system and computer program that enables easy selection of multipleobjects, distinguishing objects, enhanced interaction of objects,enhanced organization of objects, enhanced visualization of objectproperties and meta-data as well as enhanced browsing techniques. Whatis further needed is a method, system and computer program producthaving effective and coherent interaction and visualization techniquesfor virtual environment organization.

SUMMARY OF THE INVENTION

The present invention provides an improved method, system and computerprogram product for organizing and visualizing display objects within avirtual environment.

In an aspect, the present invention is a method for organizing andvisualizing display objects in a virtual environment comprising:displaying the display objects in the virtual environment, the displayobjects representing one or more collections of data; and enablingreal-time user interaction with the display objects in the virtualenvironment, wherein the user interaction with the display objects inthe virtual environment is defined by pre-determined mechanics rules.The pre-determined mechanics rules are in essence computationallysimulated physics implemented as an interface for enriched userinteraction with display objects, e.g., file icons on a virtual desktop.

A virtual environment with simulated mechanics is advantageous as itallows objects to be dragged and tossed around with the feel ofrealistic characteristics such as friction and mass, and objects cancollide and displace others. Interactions feel more continuous, analog,natural and realistic to the user, rather than the discrete, rigid,mechnical style imposed by digital computing. This allows users to usethe strategies they employ in the real world to both implicitly andexplicitly convey information about the objects they own. The presentinvention also supports the casual organization of information in amanner where users are not forced to commit to categorization, such asthe immediate naming and filing of documents. In this regard, users'spatial memory and knowledge of how things move physically in the realworld is leveraged.

In another aspect of the present invention, piles are used as anorganizational entity for desktop display objects. In general, pilesrepresent an ordered sequence display objects. In another aspect of thepresent invention, piles can be used to convey playlists in a mediacontext.

In yet another aspect of the present invention, fluid interactiontechniques are used for committing actions associated with desktopobjects.

Advantageously, the present invention provides for the integration ofinteraction and visualization techniques into a coherent interface thatprovides for the easy selection of multiple objects, distinguishingobjects, enhanced interaction and realism of objects, and enhancedbrowsing techniques, among other things.

In further aspects of the present invention, various systemimplementations are disclosed. Furthermore, a computer program productof the present invention, in one aspect thereof, is best understood as acomputer application or computer applications that when loaded on acomputer is operable to facilitate the interaction and visualizationtechniques described above.

BRIEF DESCRIPTION OF THE DRAWINGS

A detailed description of the preferred embodiments is provided hereinbelow by way of example only and with reference to the followingdrawings, in which:

FIG. 1 illustrates a virtual desktop having arbitrarily sized objects,namely a pile of photos (bottom left) and casually arranged (top left)and crumpled up (top right) windows.

FIG. 2 illustrates a LassoMenu in accordance with the present invention:on the left, the Lasso selection phase is depicted; in the centre theLasso completed is depicted when pen reaches blue circle and enterscontrol menu; and on the right a resize command selected is depicted,and the remainder of pen movement adjusts resize parameter.

FIG. 3 illustrates the Lasso'n'Cross technique for pile creation inaccordance with the present invention: on the left, documents are lassoselected; in the center, a ‘Create Pile’ icon crossed and a pile iscreated; and on the right, this action is undone by “un-crossing”.

FIG. 4 illustrates organizations examples of an aspect of the presentinvention, namely: (a) casually laid out documents; (b) tidied unpileddocuments; (c) documents made into a Messy Pile; and (d) A Tidy Pilewith widgets revealed. The widgets in clockwise order from center-topare: Fisheye, Leafer, Compression-Browse, Grid, Messy/Tidy, Fan out,Move.

FIG. 5 illustrates pile browsing layouts triggered by widgets inaccordance with the present invention: (a) fisheye; (b) leafing throughlike pages of a book; (c) Compression-Browsing higher display objects toview display objects below; (d) interpolating between Messy and Tidypositions; (e) grid browse, locked down for further manipulation withPressureLock; and (f) fan out on user drawn path.

FIG. 6 illustrates a pressure cursor in accordance with the presentinvention: (a) nonnal pressure cursor with no pressure; (b) with 75% ofmaximum pressure; (c) pen is in a position where PressureLock willtrigger additional functionality; and (d) PressureLock with 100%pressure.

FIG. 7 illustrates drag and drop insertion of an object into a pile inaccordance with the present invention: (a) drag to pile; and (b) afterinsertion.

FIG. 8 illustrates Drag'n'Cross technique for precise insertion inaccordance with the present invention: (a) user drags document andcrosses Leafer widget, pen path shown by green arrow; (b) scrub tospecify insertion point; and (c) pen is released and document inserted.

FIG. 9 illustrates a pile with display objects rotated and pulled outfor emphasis: (a) in the real world; and (b) virtually in accordancewith the present invention.

FIG. 10 illustrates using the concept of axis alignment to enforce atidier appearance in the virtual environment. The ‘shelf’ (left) wasmade by pinning up a rotation-locked display object.

FIG. 11 illustrates a flowchart depicting the steps of a pile creationaspect of the present invention.

FIG. 12 illustrates a flowchart depicting the steps of a pile browsingand manipulation aspect of the present invention.

FIG. 13 illustrates the Detailed List View visualization for browsingpiled display objects.

FIG. 14 illustrates expanding a sub-pile into Detailed List View forpiles that were automatically created based on the creation date of thedisplay objects.

FIG. 15 illustrates a number of piles that were automatically createdfrom an arbitrary selection of display objects. Display objects areorganized into sub-piles according to (a) alphabetical order and (b)creation date.

FIG. 16 a hierarchical pile whose contents are laid out in a gridarrangement and 3 sub-piles are visible. One sub-pile is furtherexpanded into the grid arrangement (b).

FIG. 17 illustrates a media display object being further in its defaultstate (a), when it is further explored (b) and when its disc begins torotate to signify it is being played (c).

FIG. 18 illustrates viewing networked users virtual environments.

In the drawings, one embodiment of the invention is illustrated by wayof example. It is to be expressly understood that the description anddrawings are only for the purpose of illustration and as an aid tounderstanding, and are not intended as a definition of the limits of theinvention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides an improved method, system and computerprogram product for organizing and visualizing display objects within avirtual environment.

In one aspect, the present invention utilizes computationally simulatedmechanics to enrich user interaction with display objects within avirtual environment, e.g., on a virtual desktop where the usermanipulates and organizes icons representing their documents, images,audio, web pages, web content, products in an online store, services inan online store, videos, software programs, file system directories,among other things.

By “mechanics”, what is meant is the behaviour of physical bodies in thereal world when subjected to forces or displacements. In particular,mechanics-based movement of objects is simulated with regard to rigidbody dynamics, mass, gravity, collisions, causality, interpenetration,frictional forces, and appearance properties, among other things. Forexample, according to an embodiment of the present invention, whendisplay objects collide they bump against and displace one another in aphysically realistic fashion. A simulated gravitational force keepsobjects on the ground.

By “display objects”, what is meant is a virtual object, such as anicon, that represents one or more collections of data. A collection ofdata can be named, i.e. a file, and can be comprised of data of anysort: text, documents, audio, images, web content, products in an onlinestore, services in an online store, videos, software programs, filesystem directories, etc.

A virtual environment with simulated mechanics allows objects to bedragged and tossed around with the feel of realistic characteristicssuch as friction and mass, and objects can collide and displace others,all according to pre-determined mechanics rules. Adding mechanics to thedesktop makes the interaction feel more continuous and analog, rather,natural and realistic than the discrete, rigid, mechanical style imposedby digital computing. This allows users to use the strategies theyemploy in the real world to both implicitly and explicitly conveyinformation about the objects they own. The implementation of mechanicsin a virtual environment also supports the casual organization ofinformation in a manner where users are not forced to commit tocategorization, such as the immediate naming and filing of documents. Inthis regard, users' spatial memory and knowledge of how things movephysically in the real world is leveraged.

The introduction of mechanics to a desktop environment makes the desktopmore lively, and offers increased degrees-of-freedom of manipulation andmovement for more expressive organizations of display objects than atraditional GUI desktop where icons are kept axis-aligned and havelittle resemblance to their physical counterparts. This mechanicssimulation has a positive and subtle effect on object placement andappearance. For example, if a few documents are casually tossed to acorner they will collide and begin to accumulate in a way that isvisually familiar to the real world. Their messy arrangement subtlyaffords an unorganized state, without the user having to explicitlyspecify it.

The present invention can be implemented in either two dimensional orthree dimensional virtual environments. Three dimensions, from theperspective of a user viewing a virtual desktop on a computer screen,for example, is preferred because there are more options in terms oforganization.

In another aspect of the present invention, piles are used as anorganizational entity for display objects. Advanced piling techniquesare described below.

In yet another aspect of the present invention, fluid interactiontechniques are implemented for committing actions associated with thedisplay objects, such as piling or unpiling. Advanced interactiontechniques are described below. A pen or a touch screen or touch pad ispreferred as a primary input device because it enhances the feeling ofrealism and directness of manipulation since objects being acted uponare visible directly under the pen tip or finger stroke. However itshould be understood that these techniques also readily apply to amouse, trackball, other touch-sensitive surfaces or various other inputdevices, as would be readily appreciated by a person of skill in theart.

It should be understood that the interaction and visualizationtechniques for computer-implemented virtual environments in accordancewith the present invention include the following aspects:

(1) Realistic Feel. Display objects on the mechanics enhanced desktopmove realistically, allowing users to leverage their knowledge of howthings move in the real world. For example, after some experimentationwith the system a user should be able to figure out that tossing displayobjects to a corner will cause them to collide and pile up, potentiallywithout explicitly learning the tossing functionality.

(2) Disable Physics as Necessary. It is preferable to leverage thebeneficial properties of the physical world, but not be overlyconstrained by or dogmatically committed to realism. When appropriate,the power of the underlying computer is exploited and the pre-determinedmechanics rules turned off or altered when the mechanics simulationproves limiting, counterintuitive, or where reality can be improved on.For example, the simulated physics aspect of the present invention canbe disabled to prevent unwanted collisions between explicitly organizeddisplay objects. When a group of display objects have been explicitlypiled by the user, the response to collisions are disabled on the piledobjects so that they do not topple over when hit by display objectsaccidentally colliding with it.

(3) Tangible, Realistic Display Objects. An aspect of the presentinvention is to provide documents that feel like tangible physicalobjects. For example, display objects representing documents can becombined with added mechanics properties such as those of paper. Thisempowers users to organize their virtual desktops in more casual, subtleand expressive ways as they do in their real workspaces. In anadditional example, storage devices could be represented by displayobjects that represent their physical form factors and mass. Thisenhances the tangibility of the object.

(4) Enhanced Interaction. The present invention is adapted for use withthe new generation of touch-based computer input devices, such aspen-based devices or touch-sensitive screens. On these devices there isoften no keyboard is available for triggering interactions. Fluidinteraction that avoids excessive point-and-click interaction ispreferred when applicable, as is exploiting the pressure sensingcapabilities of the pen while avoiding designs that are problematic forpen interaction such as small clicking targets, double clicking andclicking with the right-mouse button.

(5) Discoverable, Learnable Interface. After learning a small set ofinitial basic interaction techniques, a user should be able to discoverhow to do more complex interactions on their own. Unlike many gesturalinterfaces, it is preferable to avoid requiring the user to memorize alarge gesture vocabulary before they can use the system effectively. Thepresent invention employs existing discoverable techniques that fostersmooth transitions from novice to expert behaviour. In addition,self-revealing interaction techniques are designed by using appropriatevisual cues, and support transient easily reversible actions.

(6) Smooth Transitions. To avoid startling and confusing users, thepresent invention employs smooth slow-in and slow-out transitions forevery visual change in data representation. It is established that iseasier for users to maintain a mental model of the data across smoothtransitions and less time is spent comprehending the new datapresentation.

As discussed below, the simulation of mechanics rules can be achievedusing the PhysX™ Software Development Kit (SDK), for example. The PhysX™SDK is provided by the AGEIA Corporation, maker of the PhysX™ PhysicsProcessing Unit (PPU). PPU's are dedicated hardware that handlemechanics rules calculations required by the PhysX™ SDK, freeing the CPUfor other tasks. PhysX™ SDK provides a rigid body dynamics solverfacilitating collision detection and response, frictional forces,simulated springs, mass response, spring simulation, cloth simulation,etc. Display objects (e.g., file icons) are represented by orientedbounding boxes to the PhysX™ SDK, which for each timestep returns anupdated position and orientation for each object, according to theforces in the simulation and the rules of mechanics. User inputgenerates appropriate forces on interacted objects which in turn effectother objects in the system. A simulated gravitational force keepsobjects on the ground. The introduction of mechanics simulation to avirtual environment (e.g., the computer desktop) makes it more lively,and offers increased degrees-of-freedom for more expressiveness than atraditional GUI desktop where icons are kept axis-aligned and havelittle familiarity to their physical counterparts. This physicalsimulation has a positive and subtle effect on object placement andappearance. For example, if a few display objects are casually tossed toa corner they will collide and begin to accumulate. Their messyappearance subtly affords an unorganized state, without the user havingto explicitly specify it.

The present invention contemplates a variety of particularimplementations.

As a desktop organizational tool computer program, the present inventioncould be bundled or integrated with current operating systems such asMicrosoft WINDOWS™ or Macintosh OS X™ as replacement or add-on to thebuilt-in desktop with far more capabilities. It is noted that it ispossible to incorporate only a subset of the functionality of thepresent invention to enhance current desktops. This allows leveragingthese techniques while maintaining a more familiar interface to theuser. For instance, in Microsoft WINDOWS™ the users desktop may remainthe same but when a directory is browsed, the contents of the window mayoptionally be browsed using the techniques disclosed here, amongst theother techniques of browsing a directory contents (e.g., List view, Iconview, Details view, etc.).

3D graphics required to implement the present invention could beachieved by a system-level drawing library with built-in support forhardware accelerated rendering of 3D graphics, such as the recentWindows Presentation Foundation in Microsoft WINDOWS™ or Quartz Extremein Macintosh OS X™. Alternatively using a hardware accelerated 3Dgraphics library such as OpenGL™ or DirectX™.

Another approach would be incorporating one or more aspects of thepresent invention in a specialized version of a standard operatingsystem. For instance, TabletPC optimized versions of Microsoft WINDOWS™would greatly benefit from the pen-centric interaction techniques, sincecurrent TabletPC's simply use the version of windows designed to bedriven by a mouse, making interaction cumbersome and awkward.

Alternatively, a stand-alone deployment of one or more aspects of thepresent invention could be released installed on top of and integratedwith current operating systems. Such an application could be used inaddition to or in replacement of the current desktop. A desktop inaccordance with the present invention could be populated by the displayobjects from users previous desktop file system directory and changesmade on the enhanced desktop in accordance with the present inventionwould be reflected in the file system. Objects could be bi-directionallymoved to and from existing windows and applications to the desktop inaccordance with the present invention with drag-and-drop. In addition,activating objects (files or folders) in the desktop interface inaccordance with the present invention could launch their appropriateviewer applications.

Another stand-alone approach involves the desktop in accordance with thepresent invention workspace being completely disjoint from the existingdesktop, where documents can be freely arranged without concern for whatexists on the desktop.

It should be understood that the present invention can be applied to anyimplementation having a graphic user interface where advance interactionwith display objects is desirable, and is not limited to just personalcomputers. For example, the virtual environments in personal digitalassistants (“PDAs”), mobile phones, BLACKBERRY™ and other devices wouldbe similarly enhanced by the techniques in accordance with the presentinvention. What are required are a display means, an input device, and asuitable computing means to implement mechanics rules and otherinteraction and visualization techniques.

Furthermore, both client, web-enabled client-server and peer to peersystems are contemplated. A web-based embodiment of the presentinvention would be operable to enable the mechanics and organizationtechniques functionality to a client computer through the Internet. Peerto peer connections of any virtual environment client implementation isalso contemplated by the present invention

As discussed herein, the present invention contemplates making increaseduse of document meta-data and enhanced visualization of display objectattributes. For example, file size is mapped to mass or volume orfriction index of the display object's representation within the virtualenvironment. Other physical properties could be used as cues forconveying content information. For example, larger files might moveslower because they feel ‘heavier’ or older files appear dog-eared toshow their wear. Other physical phenomena could be employed to guide thedesigns. For example, objects could be modeled as sheets of paper thatcan be folded in interesting ways to convey information or draped overother objects. It would be also useful to be able to pin something tothe desktop like you could to the walls. PressureLock and tear awaytechniques (described below) could be used to pin and unpin displayobjects.

Gravitational fields that surround display objects can also be used toattract other similar display objects (e.g., of the same file format)situated nearby within the virtual environment. For instance, a pilefull of photos would have a greater pull on a photo being tossed by itthan a neighbouring pile of spreadsheets. A gravitational fieldimplementation would also support the escape velocity to allow displayobjects to be tossed beyond piles that may attract it. Another extensionof the physical metaphor includes using a magnet to gather like iconsfrom the desktop. For instance, a video magnet could be used to attractvideo files from a large unorganized heap of documents while maintainingcontext. Multiple magnets could be used in conjunction to see spatialrelationships.

Furthermore, in real workspaces the landmarks and layout of the deskgreatly inform how display objects will be organized. For example, wallscan have notes posted on them or shelves will be used to storedocuments. A desktop interface in accordance with the present inventioncan incorporate this in a number of ways. For instance, the layout ofuser's desk can be digitally replicated, or offer virtual templates ofstandard physical desk configurations to increase user familiarity withthe system. Alternately, variations to the desktop's layout could bemade to aid in organization, such as permanent shelves, orrecessed/raised areas for storage.

In a multiple user scenario, multiple workspaces can exist within asingle plane, which when zoomed out could facilitate the transfer ofdocuments between users by tossing them to the desired workspace.Alternatively, these multiple workspaces could be used by a single-userto management and organization of different tasks.

It should be understood that the interaction, visualization andorganizational techniques described herein are also well-suited fordesign applications, either two- or three-dimensional. For example, inthe architectural context, a designer is often required to manipulate agreat number of objects of his/her virtual desktop. A virtualenvironment equipped with physics simulation and advanced pilingtechniques would greatly improve the efficiency by which a designer cancreate and manage files. As well, the fluid pen or touch basedtechniques will enhance the feeling of realism and directness ofmanipulation since objects being acted upon are visible directly underthe pen tip, which is akin to the traditional, and often preferred, penand paper means of design.

It will be appreciated by those skilled in the art that other variationsof the embodiments and implementations described herein may also bepractised without departing from the scope of the invention. Furtherillustration of the method, system and computer program of the presentinvention is provided in the following non-limiting examples.

EXAMPLES

Note that the term “BumpTop™” is used herein to describe a computerproduct in accordance with the present invention, according to oneexample embodiment.

BumpTop™ provides users with a perspective “2½D” view onto a planardesktop surface tilted 25 degrees with respect to the camera, asillustrated in FIG. 1. This angle is preferred over a top-downperspective view (which resembles a flat desktop users are accustomedto) because users may find it difficult to distinguish the depths ofpiles and confused them with single objects.

In this example, motion is constrained to the desktop by the wallsenclosing it. The wall corners provide a place for documents to pile upon top of each other and act as a landscape feature that could aid incognitive grouping of documents. Alternatively, the user can also enablethe use of the screen boundaries as walls, off of which display objectswill bump and collide. The desktop texture has a number of circles whichact as “passive landmarks” that could aid in visually separating groupsof display objects. However, users are free to place documents anywhereon the surface.

Files (e.g., documents, text, images, videos) are represented by displayobjects whose geometry is a 3D cube squashed on one axis, and istexture-mapped on all sides so that when vertically stacked there is anindication of its type. Being able to discern information from iconedges supports a pile browsing behaviour that occurs in the real worldcalled edge browsing. Also, non-zero depth is necessary for the boundingvolumes used in the collision detection. Textual labels are optionallypresented on top of the display objects.

It is important to note that the techniques of the present invention arescale-independent and can work on any mixture of arbitrarily sizedobjects. This allows for interesting usage scenarios such as theorganization of windows or photographs, as illustrated in FIG. 1. Amixed-mode approach is possible where display objects and folders in thedesktop launch standard windowed applications. An alternative isallowing the windowed applications to benefit from the physics paradigm.

Mechanics-based movement of objects is simulated with rigid bodydynamics, collision detection, and frictional forces. When objectscollide they bump against and displace one another in a physicallyrealistic fashion. A simulated gravitational force keeps objects on theground. The introduction of mechanics simulation to a desktopenvironment makes the desktop more lively, and offers increaseddegrees-of-freedom for potentially more expressiveness than atraditional GUI desktop where icons are kept axis-aligned and havelittle resemblance to their physical counterparts. This physicalsimulation has a positive and subtle effect on object placement andappearance. For example, if a few documents are casually tossed to acorner they will collide and begin to accumulate. Their messy appearancesubtly affords an unorganized state, without the user having toexplicitly specify it.

It should be understood that BumpTop™ enables casual organization ofdocuments as one would on the surface a real, physical desk furniture,using piling rather than explicit filing as the primary organizationalstyle. In accordance with the present invention, a variety ofinteraction and visualization techniques are utilized implicitly andexplicitly for creating, manipulating and organizing piles and displayobjects within the piles.

Interaction and Visualization Techniques

An input means, such as a mouse, or preferably a pressure-sensitive penwith a single barrel button operating on a TabletPC, allows the user tointeract with the display objects in the virtual environment. Tofacilitate very lightweight interaction for the simplest tasks, the penby default allows users to move and or toss objects by touching and thendragging or flicking them with the pen in a manner similar to how onemight use a finger to manipulate a bunch of lightweight items on aphysical surface. A spring is attached from the point on the displayobject the user first clicked on, to the current cursor position. Thisallows for the free form, natural, realistic intuitive movement andtossing of display objects. More complex interactions require additionaltechniques.

LassoMenu

The interaction of display objects can be triggered by a techniquecalled LassoMenu that combines selection, command invocation, andparameter adjustment in one fluid stroke, as illustrated in FIG. 2.Users select display objects in the typical “lasso” fashion of drawing apath that encircles them. Once the lasso stroke has begun and the lassostroke has reached a minimum distance, a semitransparent circle isplaced at the beginning of the lasso stroke. The minimum distance is inproportion to the size of the semitransparent circle. If the stroke isclosed by the pen path entering the circle or by re-intersecting thelasso stroke itself, users are presented with a control menu, a markingmenu variant in which the user first selects a menu item via a simplestroke and then can choose to smoothly move the pen in a 2D fashion toadjust the value of an associated parameter.

The LassoMenu avoids the pigtail gesture that some users found difficultand was less preferred than the handle technique in the prior art. Inaddition, the LassoMenu is more fluid than the known handle techniqueswhich interrupts the stroke by requiring the pen to be lifted for themarking menu to appear. Further, there is no gesture to memorize. Theunobtrusive semi-transparent blue circle indicates additionalfunctionality, and the user is not penalized for simply exploring it aslifting the pen up before they leave the blue circle does nothing. Theinclusion of a control menu enables the fluid transition from novice toexpert functionality in that novice users can browse the menu visuallyto identify and select the desired items while experts who haveperformed the same selection numerous times in the past can simply makethe stroke in the appropriate direction without visually attending tothe menu itself. Note that the LassoMenu can, if desired, be usedsmoothly in combination with existing techniques like the handle andpigtail. Using the pigtail with the LassoMenu allows for commandinvocation without closing the lasso stroke. It will be appreciated bythose skilled in the art that while we describe much of the interactionin the invention as being triggered by the LassoMenu, it may be beinvoked by traditional mechanisms instead as well (ie, keyboard keys,menu option, etc.), without departing from the scope of the invention.

Display Object Movement

Display objects on the desktop can be dragged around, and can beattached to the pen position by a dampened spring. This is a popularmethod of interaction with physical simulations. Movement in thereal-world is smooth, where velocities gradually rise and fall insteadof the instantaneous movement found in typical GUI applications. Byincorporating this spring model into the technique, it affords a subtleeffect on the feel of the interaction, making it more lively andphysically realistic.

Another benefit of the spring is that it allows a quick flick of anobject to toss it across the screen. The display object will naturallydecelerate due to friction and will bump and displace objects in itspath appropriately. The quicker the flick, the further and moreforcefully the object will travel. Multiple objects or piles are movedand tossed in a similar fashion. When a user lasso selects multipledocuments, they are highlighted and invisible dampened springs arecreated between them with a complete graph topology. Selection springsallow the drag or toss of one document to tug along the other documentsin the selection while maintaining their relative spatial positioning toeach other. These springs are released when documents are deselected byclicking a vacant area of the desktop or starting a new selection.

The pen or touch input can also be used to ruffle through and nudgeobjects aside as if it had actual physical geometry in the workspace. Inthe case of a pen, this is accomplished by holding down the pen barrelbutton while moving it on or above the screen surface. It is noted thataccidental triggering has been known to occur with use of the barrelbutton though new pen designs which move the button further up on thepen minimize this. Alternatively the functionality can be triggered by adifferent mouse button or keyboard modifier key. The objects inaccordance with the present invention behave as if they had certainphysical properties. They are moveable, rigid, bouncy, and toss-able.These properties enable a more physically realistic environment andafford users to organize their virtual objects in more expressive ways.

Pile Creation—Lasso'n'Cross

To more explicitly organize a group of display objects piles can becreated. Piles are created by lassoing around a group of displayobjects, then crossing the ‘create pile’ icon that appears at thegroup's centroid. This technique is called Lasso'n'Cross (FIG. 3). Thistechnique allows users to fluidly select and pile objects in one stroke.Novice users will typically wait until they notice the icon beforecompleting the stroke, but as they practice making the stroke oversuccessive invocations, they transition seamlessly to expert behaviourwhere the stroke is made without waiting for the icon to appear.Lasso'n'Cross also supports undo, allowing users to undo and redo thepile creation by consecutively re-crossing the icon. Undoing can bethought of as “un-crossing the initial cross” since the stroke is undoneby making it backwards.

Lasso'n'Cross is an improvement over similar pen-based gesturescombining selection and a single action in prior art, such as a deletegesture that is triggered if the end of the stroke is inside the closedlasso. It is advantageous because it supports undo and eases therequirement of remembering a gesture by facilitating discovery amongstnovices.

By using the convex hull of the lasso stroke to indicate selecteddisplay objects (illustrated in FIG. 3) unwanted changes to theselection are avoided from the stroke portion that approaches andcrosses the Lasso'n'Cross icon. Further, to prevent accidental crossingof the ‘create pile’ icon, the icon is only visible when the centroid isnot likely to fall near the users lasso stroke. This is typically whenthe stroke is not a straight line. A heuristic is used to determine if astream of user input points is a straight line:

lasso arc length/distance between lasso endpoints>1.2

Tidying, Messy Piles and Tidy Piles

When creating a pile with Lasso'n'Cross the selected object'sorientations are tidied, vertically sorted according to their heightsand stacked into a Tidy pile, as illustrated in FIG. 4 d. The resultingpile replaces the ‘create pile’ icon at the centroid of the selectedobjects. This is smoothly animated to avoid confusing the user with aninstantaneous new representation of the objects.

Alternatively, using a single LassoMenu option on unpiled objects, theuser can choose to tidy them, create a messy pile, or create a tidy pileout of them. The option is determined by the distance between the endpoints of the stroke drawn after “Tidy/Make Pile” has been selected inthe LassoMenu. The option selected in order of shortest to longeststroke is as follows: (1) tidy documents by tightening up their posesbut do not create a pile; (2) create a Messy pile; and (3) create a Tidypile, shown in FIG. 4. These options are ranges on a continuum of strokedistances and selecting in between these ranges specifies the degree ofthe particular option. The document poses are updated live and the usercan “scrub” to create the desired arrangement. Visual feedback duringthe scrub is provided by icons that appear at the range edges. That is,the points at which a messy pile or tidy pile will be created.

A messy pile integrates some of the objects' messy pose information byinterpolating between the messy and tidy arrangements of a pile. Insteadof arbitrarily displacing display objects in the pile to achieve a messyappearance, the messy pile concept incorporates meaningful spatialinformation from the unpiled state.

The particular steps of pile creation are best understood with referenceto FIG. 11. A user first lassos two or more objects on a virtual desktopin order to select them (1). Optionally, after lassoing the objects theuser can create a pigtail gesture (2), which in turn invokes a commandrequesting whether the user would like to tidy or make a pile (7). Ifthe pigtail gesture is not created after lassoing, the user may alsooptionally use the Lasso'n'Cross technique which allows fluid selectionand piling of the objects in one stroke (3). If so, a tidy pile iscreated (6), but can be uncreated if the user uncrosses theLasso'n'Cross gesture (5). If uncrossed, the objects are released fromthe lasso, allowing the user to lasso other objects (1).

If after lassoing objects the user does neither the pigtail gesture northe Lasso'n'Cross (4), the LassoMenu is entered (B). The user is thenpresented with the “Tidy/Make Pile” option (7) which, as stated above,is determined by the distance between the end points of the stroke drawnafter “Tidy/Make Pile” has been selected in the LassoMenu (8). If thestroke length is passed the “make tidy pile” threshold (8), then a pileis created (6). If the stroke length is not passed the “no pile”threshold (9), then no pile is created (A). If the stroke length ispassed the “no pile” threshold (9), then the objects are tidied (10).

Supporting Pile Browsing with Widgets

When the pen hovers over a pile, pile widgets (FIG. 4 d) are revealedallowing the user to trigger various browsing techniques of the pile'scontents (FIG. 5). Generally, the techniques are designed explicitly tosupport real-world pile browsing behaviour observed in office workers.The Fan-Out widget spreads pile display objects like a deck of cards onthe user-drawn path, allowing pile contents to be viewed in parallel(FIG. 5 f). Leafing through pile contents much like one flips throughpages of a book is accomplished by scrubbing the Leafer widget (FIG. 5b). The Compression-Browse widget compresses display objects on one axisto reveal the display objects underneath, without moving display objects(FIG. 5 c). The standard grid layout is also offered (FIG. 5 e). Largerpiles benefit from a fisheye view (FIG. 5 a). The Messy/Tidy widget islike the inverse of the “Tidy/Make Pile” pile creation functionalitydescribed above. Seeing how piled objects were originally strewn aboutthe desktop may aid recall of pile purpose or content (FIG. 5 d).Scrubbing this widget interpolates between the messy and tidy poses andat the extreme messy pose an icon appears indicating the pile will bebroken. Another view of piled contents is the Detailed List View. Inthis view display objects are vertically arranged in a line, withfurther textual information about the display object (i.e., in the caseof files the file name, creation date and file size) displayed to theobjects right-hand side (FIG. 13).

A user clicks and drags on a widget to immediately see its impact on thelayout of the pile contents. Once the pen is released the objectssmoothly return to their piled state, facilitating quick, transientbrowsing. For more involved interactions a pile can be locked down intoany of the browsing states. This is done with the PressureLock techniquedescribed herein. Once a pile layout is locked, the widget turns into ared X (FIG. 5 e) and can be collapsed back to its original state with atap.

Hovering over a widget for some time presents a tooltip with a moredetailed description of the widget's functionality. Widgets also act ascrossing targets for the novel Drag'n'Cross technique for preciseinsertion of objects into a pile, as described later. All transitionsbetween browsing styles are animated smoothly.

Objects need not be explicitly piled before applying the browsing tools.The LassoMenu may be used to trigger browsing of unpiled objects. Forexample, it may be useful to temporarily view casually strewn objects ina grid layout to see occluded objects. Compression-browse wouldsimilarly reveal occluded display objects without disturbing displayobjects. A collection of one or more display objects may beautomatically piled into several piles according to informationregarding the display object (FIG. 15) (i.e., date the file was lastmodified, the first letter of the file name). We have found thatselecting a small number of sub-piles to represent the categories worksbest, such as 2 to 5 depending on how varied the collections data is. Inthis view the pile browsing techniques described above may further beused to allow further exploration of the display objects. Sub-piles aresmoothly animated as little as necessary to create room for the browsedpiles contents (FIG. 14).

Regional Visual Search

If a user wants to find a piled object but does not remember which pileit is in, he/she can use the browsing widgets to try and find it.However, for a large number of piles clicking widgets becomes tedious.For this situation there is the Exploding Piles functionality of thepresent invention, offering a way of visually searching pile contentsregionally. Once Exploding Piles is invoked with the LassoMenu, pilesare smoothly exploded into a grid view on hover. Moving the pen awaycollapses piles back to their original state. Exploding Piles exploitsthe rough spatial memory a user might have about what they're lookingfor. For example, if it is known a display object is in one of the pilesin the top-right of your workspace you can inspect them by pointing atthem.

Pressure Cursor and PressureLock Techniques

When users push very hard with the pen and reach the maximum pressurelevel, it acts as a trigger dubbed PressureLock which is used, forexample, to lock a pile down into a specific browsing layout or pinningobjects to the wall. Pushing the pen hard on the screen surface forpinning evokes similar actions in the real world.

To provide continuous visual feedback for the PressureLock techniquethere is provided a circular pressure cursor with an inner circle thatincreases in size with the current pressure level (FIG. 6). When thepressure level reaches its maximum, the color intensifies and theoutline turns into a bright white to indicate a PressureLock hasoccurred. When a PressureLock is possible it is indicated by the outerring turning a hollow white, enabling discovery amongst novice users.When PressureLock is used for locking down a pile browsing layout, thereis provided a pressure-based confirmation of a preview, with the pen-upbefore maximum pressure is reached being equivalent to an undo. Whenpressure input does not exist, we can use a keyboard modifier key or amouse button such as the standard right-mouse button to trigger thePressureLock interactions. This enables pressure-triggered interactionto work seamlessly with other input devices that don't provide pressureinformation (e.g., mouse, trackball, some touch sensitive screens, etc).

Adding to a Pile

In the real world one simply drops objects onto the top of a pile.Similarly, for casual and quick addition to the top of a pile we supporttossing an object towards a pile. This is implemented by a thresholddistance for piles that when approached by object(s) above a certainvelocity inside that distance, they are smoothly added to the top ofthat pile. Alternatively gravitational fields are used to attract tosseddisplay objects towards piles of similar content. For instance, a pilefull of photos would have a greater pull on a photo being tossed by itthan a neighbouring pile of spreadsheets. This also supports an escapevelocity to allow display objects to be tossed beyond piles that mayattract it. Another technique for adding to a pile allows displayobjects can be dragged on top of a pile that will highlight indicatingthat they will be added to the top on pen up. If the user drags anobject to a pile and dwells, the pile is temporarily pushed apartallowing for precise insertion of that object into any location withinthe pile. Scrubbing the pen along the side of the pile varies theinsertion point.

To avoid dwell which interrupts user flow, the present inventionprovides a more pen-centric interaction technique called Drag'n'Cross(FIG. 8). While dragging objects, users can cross through a pile widgetto use one of the browsing techniques for specific insertion. Forexample, if you drag an object and cross the Leafer widget, the objectwill be inserted at the point that you had leafed to before lifting thepen. After precise insertion, added objects slightly stick out of thepile in the direction they were added from. This indicates the recentinsertion and reminds users that further organization may be necessary.To tidy the pile again, the user can simply adjust the Messy widget.

Hierarchical Piles

Elements of the two paper processing strategies can be blended: pilingand hierarchical filing. In this hybrid technique, users can merge anycombination of piles and objects into a new pile, using the sametechniques employed to create a ‘flat’ pile out of just objects:LassoMenu or Lasso'n'Cross. The new hierarchical pile stores allinformation regarding the sub-piles and includes display objectssticking out in sub-piles. If the hierarchical pile is broken, sub-pilesare restored in their original positions with changes, like displayobject deletions, folded through. When the hierarchical pile is browsedwith one of the techniques described below, the sub-piles are seen aspiles in the laid out version (FIG. 16 a). These subpiles can further bebrowsed in the same fashion, and browsing them smoothly animates toadjust the already laid out display objects appropriately. The laid outsubpile is shaded with a dark region behind it so that its contents tobe distinguished from its parent pile (FIG. 16 b). It should beunderstood that this works recursively for any arbitrary number ofsubpiles within an aggregate pile.

Manipulation of Pile Contents

While a pile is locked down into a browsing mode via the PressureLocktechnique or its mouse or keyboard equivalent described above, one canfurther manipulate the pile contents with the LassoMenu. While addingphysics to the desktop enhances the realism, a user is not constrainedto only physically realistic interactions. For example, a user caninstantly sort piles or subselections by type or size. Deletion andduplication is also possible. To re-arrange pile order a user simplydrags objects(s) to their new location within a locked down pile.

On real desks subtle techniques are used to convey information aboutobjects in piles such as re-positioning or re-orienting certain displayobjects so they stick out. The present invention supports similarfunctionality to emphasize, distinguish and indicate separation of pileddisplay objects. Display objects in a locked down pile can bere-oriented from the LassoMenu. Alternatively, groups of display objectscan be re-positioned so they stick out (FIG. 9). Once re-positioning isinitiated the pile smoothly collapses back to its piled state so it canbe seen in context. Dragging moves objects parallel to the plane thepile sits in, to avoid changing pile order. If the objects are draggedso far that they no longer overlap any part of the pile, they are pulledout of the pile and become the active dragged selection. Note that thepen is still down and a user may fluidly proceed with other dragginginteractions from here such as insertion into a pile via Drag'n'Cross orpinning up to a wall. Dragging display objects out of piles could alsobe used to split a pile if it is too large or cumbersome.

The particular steps of pile browsing and manipulation are bestunderstood with reference to FIG. 12. A user first crosses or selects apile widget (1). Alternatively, the LassoMenu is invoked on the pile(4), providing a further means of a selecting a browsing method (5).Upon selection of the browsing method on the widget (2), PressureLock(6) can be triggered to lock down the pile for manipulation (7). Thereis a “close pile” button (8). Once locked down into a browsing modeusing the PressureLock technique, the pile contents are manipulated withthe LassoMenu (9). Objects can be shifted (10) by moving the cursorwhich alters the object's position in the pile (11). Objects can besorted be size (12), and subsets can also be sorted (13). Objects can besorted be type (14), and subsets can also be sorted (15). One or moreobjects can be deleted (16), or subsets deleted (17). Objects can alsobe moved and reoriented (18), using the cursor (19).

Enhancing Realism

Frequently accessed display objects are usually moved to the top oftheir piles, leaving less relevant material at the bottom due torepeated re-referencing. This is facilitated in the present invention bysupporting easy removal and casual addition to the top of piles viatossing.

Giving Display Objects Affordances of Paper

The physical properties of paper include being thin, light, porous,opaque and flexible. Further, these properties afford different humanactions including folding, crumpling, creasing, and pinning.

Our invention supports several manipulations that physical paperaffords, in line with an aspect of the present invention comprisingTangible, Realistic Display Objects. This is supported by freeformcreasing of a corner or folding of a document (FIG. 1). To escalate adocument to even greater importance, it can be pinned up to a wall, orby using PressureLock to create a springy joint. To remove it a usersimply pulls the document off the wall. Another technique is locking therotation of objects to make them standup on end (FIG. 10), despitecollisions. This is accomplished with a LassoMenu triggered rotation andPressureLock once the desired rotation is specified. Further, and incombination with all of the previous techniques, objects may be resized.Object density remains constant so bigger icons have more mass whilereducing size reduces mass. Bigger documents are not only more visuallynoticeable but behave as though they are more important, displacingsmaller display objects when moved and being difficult to move whenbumped by smaller, lighter display objects.

To de-emphasize display objects a user can crumple them up (FIG. 1).This aspect provides an in-between state for display objects whoseutility is questionable but are not quite ready for deletion. Crumplingis done from the LassoMenu which specifies a 2D parameter for distortingthe object mesh. For example, a long quick stroke results in a tightlycrumpled document.

When computer programs or services first launch, users are oftenpresenting with a “splash screen” that contains the program or servicesbranding information, amongst other things. In this invention we texturemap “splash screen” onto simulated cloth, which can be teared open bythe users cursor using mechanics rules. This is meant to be akin toopening a present in the real world, and slowly reveals the virtualenvironment presented behind the cloth. The cloth that is torn awayfalls into the virtual environment due to gravitational forces and fadesaway after a short time (5 seconds). After more than some percentage(e.g. 60%) of the virtual environment is visible the splash screen maydisappear. Further, a button is presented so that at any time the usermay skip the tearing process, as it is simply meant to enhance enjoymentand realism within the invention, and not be inefficient.

Paper-like attributes are appropriate for display objects representingcontent that in the real world is often found on paper. Such displayobjects include documents, text, photographs, emails, web pages, etc.For other data types, such as media representing audio or video, otherattributes are appropriate. In one aspect of the invention, we representmedia display objects as consumer media products such as DVDs or CDs.The unactivated media display object appears with similar proportions tothose in FIG. 1. One animation for further exploring the displayobject's contents (ie, to get more information about the particulardisplay object, or to initiate playing of media display objects) is theflipping open of the virtual cover of the display object (FIGS. 17 a, 17b). For media display objects this is visually familiar to opening a CDcase. Further, an object which resembles a compact disc that is alsotexture-mapped with a relevant image smoothly rises from the “case” andspins if the media display object is played (FIG. 17 c).

Animating Introduction of New Content

To introduce new content to the system we animate it in a user friendlyfashion by “Dropping from the Sky”. “The sky” is better defined as theposition of the camera looking into the virtual environment. Objectsdrop in the direction the camera's view onto the virtual environment.Gravitational forces pull the falling display objects towards thevirtual environment's surface. Mechanics rules govern the motion thefalling display objects experience. This technique enhances realism andprovides a user friendly metaphor for introducing new content to theuser. For example, new emails may drop from the sky onto a particularregion on the virtual desktop environment when they arrive, accumulatinginto a pile that visually reflects that their unorganized state. It alsoestablishes a space where the user can expect to find new displayobjects of a particular type, leveraging spatial memory. Another exampleis If a duplicate of a display object is made, its duplicate can dropfrom the sky on top of or close to the source display object. In thiscase location for the drop is automatically determined by theduplication command. Another example is presented in the web browsercontext. Current web browsers such as Mozilla FIREFOX™ and MicrosoftINTERNET EXPLORER™ 7.0 allow multiple webpages to be browsedsimultaneously using “tabs”. When a new tab is created in theseinterfaces it is placed in a one dimensional row of the existing tabs.In our invention, the tab would “drop from the sky” into the virtualenvironment, and could be organized as any other display object. Thisallows users to use three dimensional spatial memory and mechanics rulesto enhance organization and management of display objects.

This technique may further be extended for collections of data thatbecome available to the computing environment. When said data collectionbecomes available a display object representing the proportions andappearance of the physical form factor of the data collection can “Dropfrom the Sky” into the virtual environment. For example, if an AppleIPOD™ is connected to the computer, a 3 dimensional display object theresembles the Apple IPOD™ will land into the virtual environment. As aperson skilled in the art would understand, these data collections couldbe any type of collection of data. For example, connecting an externalUSB storage device to the computing environment, or networked filesresiding on a remote machine to which a connection was just made. Totransfer display objects to the data collections represented by displayobjects, the user may toss it towards the general direction of the datacollection display object and it will be added according to the tossingalgorithm provided for adding to a pile provided above.

Viewing Other Virtual Environments

Users of multiple networked virtual environments may visualize thecontents of other user's virtual environments that are connected vianetwork. To enhance privacy, each user may specify a subset of displayobjects to make visible to other networked users, or rules fordetermining how which display objects to share with other networkedusers on their remotely displayed virtual environment. (i.e. displayobject icons may be visible while filenames are obscured to enhanceprivacy). In a networked environment, this can facilitate awarenessbetween collaborators. Further, tossing can be used as an interactiontechnique to transfer files between these networked virtual environmentsusing the algorithms described above. To smoothly transition to view ofnetworked user's virtual environments, the camera can zoom out of theusers personal environment to a view such as that presented in FIG. 18.The networked virtual environments can exist in a plane, and may bere-arranged according to mechanics rules as other display objects in thesystem.

Enhanced Search

When executing a search query, display objects representing the searchresults will “drop from the sky” into the virtual environment, asdescribed above. This enhances the physicality and tangibility of thesearch. Results can further be organized in the same flexible ways asother display objects in the invention. The search query could beexecuted on any combination of information sources. For example, theusers local hard disk, the internet's webpages, photographs, audio,videos, etc. In the example of web browsing, popular search engines suchas GOOGLE™ or YAHOO™ typically represent search results as a linearlisting of short text-based summaries. Our invention allows the resultsof a web search to be browsed in parallel as thubmanils of the webpagetexture mapped onto display objects in the virtual environment. Browsingvisual results in parallel enables better user performance of findingthe information the are looking for due to increased informationalbandwidth, and use of spatial memory to facilitate the organization,management and sense-making of search results.

We also embody other search result meta-data, such as relevance tomechanics properties of the display object. For example, more relevantsearch objects are represented as slightly larger and heavier than thosethat are less relevant. To represent related search results, we canpresent the group of related search results as piles. In these piles thetop element would the original result from the search query. These pilescan then be browsed using the techniques described above. The pilerepresentation also allows the user to visualize how much relatedcontent is available.

Enhanced Tagging

Our invention supports enhanced techniques for tagging display objects.Display objects may be tossed around to create loose arrangements of thedisplay objects. Tossing is advantageous because the mouse needs totravel less distance to toss an object, since it need not move exactlyto a desired point, instead a general direction can be specified. Inaddition, tossing is more casual and as such encourages facilitatesloose arrangements to be created before a high level structure of theinformation may be known. Once loose arrangements are created,Toss'n'Tag can be activated which detects white space between displayobjects and allows tag's for each grouping to be specified.Alternatively, the user may choose to associate spatial regions withspecific tag(s) before beginning to toss things. These regions canpartially overlap since tags are not mutually exclusive. If the regionsfully overlap they are collapsed into a single region with multipletags. The regions can be defined by encircling a region and theninvoking the LassoMenu and selecting “Create Tag Region”. Yet anotherway to tag documents includes creating tags that are visualized andinterated as other display objects in the system. They can then betossed at other display objects to tag them. When tags are piled theycan be tossed as a group. These techniques apply to various displayobject types, and are highly relevant to images, documents, web pages,web content amongst others.

Polite Physics

The physical simulation can sometimes prove disruptive, and occasionallyit can be disabled. For example, when display objects are in a messy ortidy pile and if physics are enabled, the collision detection ofperfectly touching display objects can cause numerical instability,jittering and unnecessary movement by piled display objects. Also,dragged icons can knock over piles.

Further, un-restricted mechanics allows six degrees of freedom in thepotential positions of objects. This added freedom and expressivenessalso affords more ways to “make a mess”. Therefore, in a further aspectof the present invention, all display objects can remain axis-aligned.Collisions are no longer physically accurate but objects are easilyreadable remaining properly oriented, more closely resembling modem GUIdesktops. In this mode the desktop seems aesthetically tidier but looksand feels mechanical (FIG. 10). Alternatively, a small amount ofoff-axis rotation can be allowed to make the appearance less rigid whilemaintaining readability and preventing objects from becoming too messy.We allow a fixed number of degrees of rotation about the axis that isnormal to the virtual environment's desktop surface.

Implementation

In a particular implementation, BumpTop™ runs in real-time on a ToshibaM200 TabletPC with a 1.6 Ghz CPU, 1 GB RAM and an NVIDIA GeForce™ FX Go5200 graphics card. The example, the software is written with C++,OpenGL and GLUT.

The mechanics, e.g., the rigid body dynamics, physical simulation andcollision detection rules, are provided by the AGEIA PhysX™ SDK.

User Evaluation

To evaluate the implementation of the present invention a qualitativeuser study was conducted. Six participants (2 female, 4 male withcomputer skills ranging from novice computer users to pen-interfaceexperts) participated in videotaped think-aloud sessions lasting an houreach. This consisted of a 3 min introduction, 10 min “discovery” periodwhere users explored the system, followed by instruction on theremaining functionality they didn't discover. Finally 29 tasks wereperformed that could be completed using multiple strategies. Post-studywritten and verbal questionnaires were also completed.

The results were positive. Participants were able to discoverfunctionality on their own and became comfortable and proficientaccomplishing most tasks. Questionnaire responses confirmed that:techniques were easy to learn (4.7/5), users were able to accomplishwhat they trying to do (4.4/5), users liked the software (4.7/5), andsoftware felt familiar (4.5/5). Techniques like tossing were foundempowering as they allowed leveraging of real-world knowledge. Manyparticipants found the interface playful, fun and satisfying.

Lasso'n'Cross was the preferred technique for pile creation. Most usersquickly became comfortable with it and several stated that creating andbrowsing piles with Grid, Leafer or Fanout were amongst their favouriteinteractions.

With respect to the discoverable, learnable interface, users were ableto complete approximately 88% of tasks without extensive training.Participants used the 10 min “discovery” portion of the experiment indifferent ways. Some experimented with the movement of objects andspatial layouts. All experimented with the pile widgets to invoke thevarious browsing methods and discovered interaction techniques notexplicitly mentioned in the introduction. In addition, some participantsemphatically stated that if they were given the tasks again they couldeffortlessly complete them.

With respect to realistic feel and enjoyable user experience, during thediscovery period and idle time between tasks users were seen playfullytossing or rearranging display objects or watching the results ofcollisions. This playfulness also translated to users becomingproficient at arranging display objects with subtle and precisemovements. For example, within minutes one participant was delicatelybalancing a document on top of another document pinned up to the wall.One participant carefully arranged icons on edge and toppled them overlike dominoes. This behaviour suggests successful leveraging of realworld knowledge of movement. Tossing was preferred to dragging forinsertion by 4 of 6 users. Pressure based techniques were learned andused with little error.

1. A method for organizing and visualizing display objects in a virtualenvironment, the method comprising: (a) displaying the display objectsin the virtual environment, the display objects representing one or morecollections of data; and (b) enabling real-time user interaction withthe display objects in the virtual environment, characterised in thatthe user interaction with the display objects in the virtual environmentis defined by pre-determined mechanics rules.
 2. The method of claim 1further characterised in that the virtual environment is a desktop andthe display objects are icons representing files, directories andprograms.
 3. The method of claim 1 further characterised in thatattributes of the display objects define the user interaction accordingto the mechanics rules.
 4. The method of claim 3 further characterisedin that the attributes of the display objects are associated with theone or more collections of data.
 5. The method of claim 4 furthercharacterised in that appearance of the display objects are defined byattributes of the one or more collections of data.
 6. The method ofclaim 1 further characterised in that the mechanics rules includesimulation rules relating to gravity, position of the display objects,mass of the display objects, velocity of the display objects, frictionbetween the display objects, collisions between the display objects orrigidity of the display objects.
 7. The method of claim 1 furthercharacterised in that the virtual environment is a virtual twodimensional or virtual three dimensional desktop.
 8. The method of claim1 further characterised in that the one or more collections of data aredigital files selected from the group consisting of text, documents,images, audio, web pages, web content, web search results, products inan online store, services in an online store, videos, software programsand file system directories.
 9. The method of claim 1 furthercharacterised in that the user interaction includes piling of thedisplay objects.
 10. The method of claim 9 further characterised in thatthe piling includes pile-pile interactions, intra-pile interaction, pilewidgets, hierarchical piles, transition between unpiled and pileddisplay objects, and piling of arbitrarily sized display objects. 11.The method of claim 1 further characterised in that the user interactionincludes enhanced interaction and visualization techniques using aninput device.
 12. The method of claim 11 further characterised in thatthe input device is a touch-sensitive input device such as a digital penor a touch-sensitive surface.
 13. The method of claim 11 furthercharacterised in that the interaction and visualization techniquesinclude LassoMenu, Lasso'n'Cross, Crossable Pile Widgets, Display Objecttossing, Pressure Widgets, PressureLock, Drag'n'Cross, Dwell'n'Scrub,and Exploding Piles.
 14. A system for organizing and visualizing displayobjects in a virtual environment, the display objects representing oneor more collections of data, the system comprising: (a) a display means;(b) an input device; (c) a computer; and (d) an application loaded on tothe computer, the application being operable to provide instructions tothe computer that: (i) display the display objects in the virtualenvironment, the display objects representing one or more collections ofdata; and (ii) enable real-time user interaction with the displayobjects in the virtual environment, characterised in that the userinteraction with the display objects in the virtual environment isdefined by pre-determined mechanics rules.
 15. A system for organizingand visualizing display objects in a virtual environment, the displayobjects representing one or more collections of data, the systemcomprising: (a) a server computer; (b) an application linked to theserver computer, the application being operable to provide instructionsto the server computer that: (i) display the display objects in thevirtual environment, the display objects representing one or morecollections of data; and (ii) enable real-time user interaction with thedisplay objects in the virtual environment, characterised in that theuser interaction with the display objects in the virtual environment isdefined by pre-determined mechanics rules.
 16. A computer program fororganizing and visualizing display objects in a virtual environment, thedisplay objects representing one or more collections of data, thecomputer program comprising: (a) a computer readable medium bearingsoftware instructions; and (b) the software instructions for enablingthe computer to perform predetermined operations, the predeterminedoperations including: (i) displaying the display objects in the virtualenvironment, the display objects representing one or more collections ofdata; and (ii) enabling real-time user interaction with the displayobjects in the virtual environment, characterised in that the userinteraction with the display objects in the virtual environment isdefined by pre-determined mechanics rules.